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ABSTRACT 


This  study  addresses  the  issues  concerning  the  upgrade  and  reuse  of  computer 
simulation  models  and  presents  a  comprehensive  methodology  —  The  Fidelity 
Enhancement  Process  —  for  conducting  a  model  upgrade.  Recent  advances  in  software 
technology  --  specifically  object-oriented  programming  and  open  architectme  system 
development  —  have  made  this  process  feasible  and  provide  xmprecedented 
opportunities  for  model  reuse.  The  Fidelity  Enhancement  Process  was  developed  and 
applied  to  the  Marine  Corps  Communication  Architectme  Analysis  Model  (MCCAAM) 
during  its  upgrade.  MCCAAM  simulates  Marine  Air  Ground  Task  Force  (MAGTF) 
single-channel  communications  architectures.  MCCAAM  was  modified  to  evaluate 
architectvu-e  performance  under  different  allocations  of  next-generation  radios  to  units 
in  the  MAGTF,  where  the  performance  of  an  allocation  was  tactically  driven. 
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I.  INTRODUCTION 


A.  STATEMENT  OF  THE  PROBLEM 

In  the  current  state  of  computer  simulation,  a  model’s  ability  to  perfectly 
replicate  the  system  being  modelled  is  hmited  by  time:  a  hardware  based  constraint. 
This  limitation  is  continuously  being  reduced  by  advancing  technology,  which  provides 
ever  greater  computing  speed  and  quicker  memory  access.  At  the  same  time,  software 
design  improvements  have  given  us  the  ability  to  modify  and  reuse  existing  models  to 
meet  new  requirements. 

Combined,  these  capabilities  place  great  power  in  the  hands  of  the  analyst  and 
give  him  a  wide  range  of  options  for  improved  model  design  and  expanded  model 
usage.  Of  special  interest  is  the  ability  that  this  increased  power  gives  the  analyst  to 
upgrade  existing  models  —  to  make  good  models  even  better.  The  big  question 
becomes:  how  do  we  determine  where  best  to  apply  our  expanded  capability  to  achieve 
this  goal? 

It  seems  appropriate  to  focus  on  areas  of  the  model  that  were  previously  limited 
or  ignored  because  of  hardwsire  or  software  constraints.  It  also  makes  sense  to  update 
the  model  to  reflect  any  changes  in  the  real  world  system  that  it  represents;  this  may 
require  changing  the  model’s  parameters  or  structure.  The  exploration  of  new  model 
uses  may  also  merit  some  of  the  available  power.  How  do  we  choose  from  among  these 
alternatives?  The  extent  of  the  improvements  to  apply  to  each  submodel  is  still 
another  complicated  decision.  It  appears  that  all  of  these  issues  must  be  addressed  to 
effectively  exploit  the  available  power.  The  need  for  a  coherent,  scientific  method  to 
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choose  which  submodels  to  upgrade  is  apparent,  but  there  is  no  methodology  available 
today  that  addresses  these  issues  in  an  organized  manner. 

B.  PURPOSE  AND  SCOPE 

The  purpose  of  this  thesis  is  to  develop  a  structured  methodology  for  upgrading 
an  existing  model.  The  importance  of  such  a  methodology  can  be  measiu-ed  in  terms 
of  the  time  and  money  saved  by  not  developing  new  models.  The  methodology 
concentrates  on  the  questions  posed  earher  including:  which  parts  of  the  model  should 
be  improved?  and  how  much  improvement  does  each  part  need? 

In  scope,  this  thesis  is  limited  to  developing  a  generic  methodology  and  applying 
it  to  upgrade  a  single  model  —  the  Marine  Corps  Commvmication  Architecture  Analysis 
Model  (MCCAAM)  —  which  was  developed  at  the  Naval  Postgraduate  School  by  a  team 
of  analysts,  including  the  author.  The  model  simulates  Marine  Air  Groimd  Task  Force 
(MAGTF)  single-channel  communications  architectures.  MCCAAM  was  modified  to 
evaluate  architecture  performance  under  different  allocations  of  next-generation  radios 
to  units  in  the  MAGTF,  where  the  performance  of  an  allocation  was  tactically  driven. 

C.  APPROACH 

New  software  technology  —  specificaUy  object-oriented  programming  and  open 
architecture  (both  described  below)  —  have  provided  unprecedented  opportunities  for 
model  reuse.  It  has  become  easier  (and  perhaps  cheaper)  to  build  and  try  model 
improvements  than  to  determine  a  priori  which  improvements  are  worth  pursuing. 
This  thesis  seeks  to  exploit  these  software  innovations  by  presenting  an  organized 
methodology  for  identifying  the  model  enhancements  that  might  be  worthwhile, 
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buUding  and  evaluating  prototypes  of  these  enhancements  and  retaining  only  those 
deemed  worthy. 
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n.  BACKGROUND 


A.  DEFINITIONS 

Before  exploring  the  background  material,  some  defmitions  are  provided  to  bind 
the  concepts  and  ideas  that  follow. 

A  model’s  level  of  Hdelity  is  the  degree  to  which  the  model  produces  the  same 
outcomes  as  the  tangible  physical  system  it  represents.  Therefore  a  model  with 
infinite  fidelity  would  produce  results  identical  to  those  of  the  actual  system. 

Aggregation  is  the  extent  to  which  a  group  of  things  in  the  real  world  have 
been  consolidated  in  the  model.  A  model  that  depicts  an  army  in  conflict  as  a  single 
entity  (object)  is  highly  aggregated,  whereas  the  depiction  of  500,000  unique  soldier 
objects  represents  total  disaggregation. 

Model  resolution  is  the  degree  to  which  submodels  are  disaggregated. 
Resolution  is  the  generic  level  of  disaggregation  within  a  submodel.  Increasing 
resolution  means  replacing  simple  decision  logic  with  more  complex  logic,  using  more 
source  data,  including  more  objects,  or  simply  improving  approximations  at  the  cost 
of  computational  performance.  Generally,  a  high  resolution  model  also  has  high 
fidelity. 

B.  OBJECT  ORIENTED  SIMULATION 

Object  oriented  simulation  (OOS)  provides  a  rich  and  easily  understood 
environment  for  building  computer  models  of  real  world  systems.  MCCAAM  was 
written  in  MODSIM 11,  a  general  purpose,  modular,  high-level  programming  language 
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that  provides  direct  support  for  object-oriented  prograniiniiig[Ref.  1].  The 
following  discussion  of  object  oriented  simulation  uses  MODSIM  11  terminology. 

The  modular  structure  of  OOS  directly  supports  model  reuse  by  allowing 
programs  to  be  constructed  from  library  modules.  Each  library  module  conteiins  a 
Definition  Module  outlining  the  t3T)e  declarations  and  an  Implementation  Module 
that  includes  all  of  the  executable  code  for  the  methods  and  procedures.  This 
structure  enables  the  developer  to  easily  improve  or  modify  one  module  with  minimal 
impact  on  the  remainder  of  the  model  and  minimal  additional  development  costs.  The 
modular  design  of  library  modules  and  the  objects  they  describe  provide  tremendous 
potential  for  comprehension  and  eventual  reuse  by  the  user  as  well  as  the  designer. 

An  "object"  combines  a  data  record,  which  describes  the  state  of  the  object,  with 
procedures  that  describe  its  behaviors[Ref.  1].  The  procedures  are  discussed  first. 
They  are  called  methods,  £md  they  describe  the  actions  that  the  object  can  perform. 
The  ASK  METHOD  in  MODSIM  II  is  equivalent  to  a  procedure  call  in  most 
languages:  the  actions  are  executed  immediately,  without  passing  any  simulation  time. 
The  TELL  METHOD,  on  the  other  hand,  is  executed  asynchronously:  the  simulation 
continues  for  some  time  after  a  TELL  METHOD  has  been  called  and  during  its 
execution.  This  allows  a  TELL  METHOD  to  pass  simulation  time  as  a  part  of  its 
actions.  While  the  object’s  methods  represent  its  actions,  its  state  is  reflected  by  its 
fields. 

An  object’s  fields  are  much  like  those  of  a  normal  record  structure  except  that 
they  can  only  be  modified  by  the  object’s  own  methods.  This  enables  the  model 
developer  to  exercise  total  control  over  the  changes  made  in  these  fields.  Problems 
become  easier  to  identify  since  they  must  be  in  that  object’s  own  methods.  Although 


5 


these  values  can  not  be  changed  by  other  objects,  they  can  be  "read'  by  other  parts  of 
the  program. 

This  reference  to  other  parts  of  the  program  brings  up  questions  about  the 
structure  of  the  program.  The  single  object  described  above  is  merely  an  object  type 
with  specified  fields  and  methods.  The  object  type  serves  as  a  template  or 
specification.  Object  instances  are  created  from  it,  dynamically,  during  the 
simulation.  Once  an  object  insteince  is  created,  its  methods  can  be  invoked  by 
messages  from  other  objects  that  ask  it  to  perform  its  methods. 

After  an  object  t3q)e  has  been  defined  by  its  library  modules,  new  types  can  be 
evolved  from  it.  Each  descendent  in  the  resulting  hierarchy  can  add  its  own  fields  and 
methods  to  those  of  its  ancestors  or  modify  an  inherited  method.  Thus,  if  we  take  a 
collection  of  objects  which  share  Vehicle  Object  as  their  ancestor  and  ask  each  to 
refuel,  the  Car  Object  might  take  on  unleaded  gas,  the  Truck  Object  diesel  fuel  and 
the  Mule  Object  would  eat  hay[Ref  1].  The  capability  of  performing  different  actions 
with  the  same  command  is  referred  to  as  polymorphism.  Combined  with  inheritance, 
it  forms  a  solid  foundation  for  reusing  these  object  types. 

C.  OPEN  ARCHITECTURE 

In  addition  to  OOS,  a  second  nuyor  software  technological  advance  is  the  move 
toward  open  architecture  system  development.  Because  of  the  new  degree  of 
standardization  produced  by  open  architecture,  models  are  portable  between 
computing  architectures  to  an  unprecedented  degree.  The  term  open  architecture 
implies  that  some  degree  of  standardization  has  been  achieved  in 
•  operating  systems, 


6 


•  graphical  user  interfaces, 

•  data  base  management  interfaces, 

•  network  operations  and  protocols,  and 

•  interfaces  to  presentation  graphics  programs. 

Open  architecture  provides  the  potential  for  model  migration  to  improve  performance 
or  to  realize  any  necessary  capability  upgrades.  Reimplementing  existing  models  on 
new  architectures  wdll  no  longer  require  developers  to  change  the  model’s  code.  This 
alone  represents  a  tremendous  savings  in  simulation  effort  that  can  be  applied  to 
upgrading  existing  models  rather  than  recoding  models  to  support 
migration. [Ref  2] 

D.  MCCAAM 

The  Marine  Corps  Communication  Architecture  Analysis  Model  (MCCAAM) 
simulates  Marine  Air  Ground  Task  Force  (MAGTF)  FM  single-channel  radio 
communication  architectures.  The  model  uses  a  workload  paradigm  of  Marine  Broad 
Operational  Tasks  (MBOTS),  Broad  Operational  Subtasks  (BOSTs)  and  Message 
Exchange  Occurrences  (MEOs).  This  framework  has  been  fitted  to  all  of  the  standard 
message  traffic  within  the  Marine  Corps[Ref  3]. 

An  MBOT  encompasses  a  broad  mission  area  and  contains  related  tasks  such  as 
the  MBOT  Artillery  Call  For  Fire.  Each  MBOT  is  further  broken  down  into  BOSTs, 
which  represent  specific  tasks  that  are  executed  by  units  of  specific  types.  For 
example,  the  Standard  Call  For  Fire  is  one  of  the  BOSTs  contained  in  the  MBOT 
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Artillery  Call  For  Fire\  it  is  initiated  by  a  Battery  Forward  Observer.  Each  HOST  is 
made  up  of  a  set  of  precedence  constrained  communications  requirements,  its  MEOs. 

An  MEO  specifies  the  unit  types  of  the  receivers  as  well  as  the  net  t3q)e  used  for 
its  transmission.  The  first  MEO  of  the  Standard  Call  for  Fire  is  a  transmission  from 
the  Forward  Observer  to  the  Battalion  Fire  Direction  Center  on  the  Battalion’s 
Conduct  of  Fire  net.  This  traffic  structure  allows  the  model  to  generate  realistic, 
interdependent  message  traffic.  [Ref.  4] 

E.  RISK  MANAGEMENT 

The  early  identification  of  risk  areas  is  crucial  to  the  successful  completion  of  any 
software  development  effort.  Risk  areas  encompass  logic,  algorithms,  data,  and  their 
associated  assumptions. 

The  importance  of  risk  management  to  mihtary  modelling  is  documented  in  the 
DoD  Standard  on  Software  Development,  which  calls  for  the  documentation  and 
implementation  of  procedures  for  risk  management[Ref  5].  The  Risk 
Management  Plan  provides  a  useful  framework  for  overcoming  mqjor  sources  of 
program  risk. 

A  Risk  Management  Plan  ensures  that  each  project  makes  early  identification 
of  its  top  risk  eireas.  These  risk  areas  include  potential  cost  and  schedule  problems  as 
well  as  the  technical  risks  mentioned  earlier.  It  is  important  to  develop  a  strategy  for 
resolving  these  risk  areas  early  in  the  development  process.  In  addition,  continuing 
emphasis  should  be  maintained  through  periodic  reviews  and  the  resolution  of  new 
risk  areas  as  they  surface.  Proper  use  of  risk  management  will  ensure  the  appropriate 
focus  on  early  protot3q)ing,  simulation,  key  personnel  staffing  measures  and  other  risk 
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resolviBg  techniques.  This  risk-driven  approach  helps  the  developer  avoid  problems 
that  might  otherwise  jeopardize  a  successful  model  upgrade[Ref.  6]. 
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in.  FORMULATION  OF  THE  PROCESS 

A.  THE  FIDELITY  ENHANCEMENT  PROCESS 

The  Fidelity  Enhancement  Process  is  a  risk-driven  approach  to  increasing  the 
resolution  of  an  existing  simulation  model.  The  use  of  OOS  and  open  architecture 
provide  the  flexibihty  needed  by  the  developer  to  efficiently  upgrade  an  existing  model 
with  this  process.  As  a  result,  the  process  is  directed  primarily  toward  models  that 
have  been  implemented  in  OOS  environments  that  support  open  architecture. 

Models  that  do  not  meet  these  criteria  present  limited  opportunities  for  reuse. 
The  model’s  lack  of  flexibility  is  detrimental  to  its  reuse  and  may  preclude  any 
upgrade  whatsoever.  In  fact,  reimplementing  these  models  in  the  desired  format  may 
not  be  possible  due  to  the  general  incompatibUity  between  most  high  level  languages 
and  object  oriented  programming  languages.  Compatibility  problems  between  differing 
OOS  environments  may  also  preclude  language  changes.  However,  incorporating  a 
newer  version  of  the  current  programming  language  or  a  compatible  graphics  package 
are  valid  changes  that  can  be  implemented. 

The  five  stage  Fidelity  Enhancement  Process  is  a  comprehensive  methodology 
for  upgrading  existing  computer  simulation  models.  It  is  formulated  for  simulations 
that  produce  a  decision  from  a  finite  set  of  alternatives.  The  stages  are  executed 
consecutively  as  portrayed  in  Figure  1. 
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Figure  1  The  Fidelity  Enhancement  Process 


B.  STAGE  1  -  THE  MODEL  ASSESSMENT 

The  model  assessment  stage  establishes  the  foundation  and  limits  of  the  fidelity 
enhancement.  The  current  capabilities  of  the  model  make  up  its  foundation,  and  the 
circumstances  which  motivate  the  upgrade  establish  the  limits.  These  limits  are 
either  hardware-driven,  model-driven  or  a  combination  of  both.  Before  exploring  the 
limits,  it  is  important  that  we  ensure  the  foundation  is  sound. 
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The  first  step  of  the  model  assessment  updates  the  risk  areas  within  the  current 
model.  Risk  areas  encompass  logic,  algorithms,  data,  and  their  associated  assumptions. 
These  risk  areas  were  used  by  the  model  developer  to  justify  the  model’s  current  level 
of  resolution.  Although  they  were  acceptable  when  the  model  was  delivered,  new  data, 
requirements  or  standards  may  invalidate  key  assumptions  that  were  made  or  logic 
that  was  used.  Any  discrepancies  must  be  addressed  at  this  point  to  ensure  a  strong 
foundation  prior  to  setting  the  model  upgrade  limits. 

With  the  foundation  in  place,  the  model  upgrade  limits  are  determined  hy 
analyzing  the  events  which  generated  the  need  for  a  better  model.  Improvements  in 
the  computer  hardware  used  by  the  model  are  the  primary  force  behind  a  hardware- 
driven  upgrade.  In  this  case,  the  user’s  primary  goal  is  to  effectively  utilize  the 
increased  capability.  Closely  related  is  the  desire  to  migrate  the  model  to  a  larger 
machine,  such  as  a  move  from  a  PC  to  a  workstation.  In  either  case,  the  hardware 
issue  becomes  one  of  known  dimensions,  which  are  specified  by  the  end  user. 

Model-driven  upgrades  involve  the  addition  of  specific  capabilities  or  the 
enheincement  of  existing  capabUities.  This  case  is  the  most  likely  scenario,  because 
historically  simulation  models  have  focused  on  specific  problems  under  specific 
conditions.  The  need  to  solve  a  related  problem  under  the  same  or  changed  conditions 
presents  an  opportunity  for  a  model-driven  upgrade.  The  limits  of  this  upgrade  may 
encompass  both  software  and  hardware  issues. 

This  combination  emphasizes  the  flexibility  of  open-architecture.  Although  the 
hardware  currently  supporting  the  model  may  offer  some  potential  for  expansion,  the 
option  may  exist  to  migrate  to  a  more  capable  machine.  This  combined  option 
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presents  the  opportunity  to  maximize  the  number  of  potential  enhancements  while 
holding  down  any  resultant  degradation  of  model  performance. 

C.  STAGE  2  -  FIDELITY  ENHANCEMENT  REQUIREMENTS 
Requirements  development  begins  with  the  definition  of  possible  upgrade 

requirements.  These  represent  both  the  end-user’s  "wish  list"  and  the  developer’s 
vision  of  the  next  version  of  his  model.  Both  may  include  new  or  modified  user 
interface  requirements  as  well  as  additional  model  capabilities.  These  needs  are 
consolidated  into  a  requirements  list,  which  is  developed  jointly  by  the  end  user  and 
developer.  It  includes  all  of  the  proposed  model  enhancements  and  their  risk  areas. 

Once  all  the  requirements  have  been  identified,  the  steps  required  to  implement 
each  of  them  are  outlined  by  the  developer.  'They  include  the  changes  to  the  model 
for  each  requirement  and  their  impact  on  the  associated  risk  areas.  'This  information 
is  used  by  the  developer  as  he  formulates  the  specific  modifications  needed  to  add  the 
proposed  enhancements  to  the  existing  model.  These  modifications  may  include  the 
addition  of  new  modules  or  objects  to  the  model  as  well  as  the  modification  of  existing 
modules. 

D.  STAGE  3  -  PROTOTYPING 

The  fundamental  benefit  provided  by  this  model  upgrading  process  is  the  ability 
to  incorporate  enhancements  while  minimizing  the  changes  required  to  the  current 
model.  This  integration  of  enhancements  is  accomplished  through  prototyping. 
Although  the  word  "prototype"  brings  to  mind  the  experimental  version  of  a  system 
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used  during  preliminary  design  work,  its  potential  as  a  tool  for  fidelity  enhancement 
goes  well  beyond  that  limited  view. 

The  protot3T)ing  necessary  for  the  Fidelity  Enhancement  Process  is  strawman 
prototyping,  which  provides  a  surrogate  system  that  can  be  investigated  to  improve 
the  design  of  the  eventual  upgraded  system[Ref.  7].  Each  enhancement  on  the 
requirements  list  is  integrated  into  the  existing  model  in  such  a  way  that  it  can  be 
turned  on  and  off  with  software  switches.  This  enables  the  developer  to  assess  the 
impact  of  each  enhancement  combination  on  the  model’s  overall  performance. 

E.  STAGE  4  -  FTDEUTY  ANALYSIS 

Once  the  prototyped  enhancements  are  in  place,  the  developer  must  assess  the 
costs  and  benefits  of  his  new  enhanced  model.  The  underlying  assumption  of  fidelity 
analysis  is  that  by  increasing  model  resolution,  the  model’s  fidelity  must  also  increase. 
While  the  result  of  increased  model  fidelity  is  a  better  model  providing  better  answers, 
the  costs  incurred  by  fidelity  enhancement  must  also  be  addressed. 

1.  Fidelity  Costs 

Any  increase  in  model  fidelity  produces  a  corresponding  decrement  to  the 
execution  of  the  model  in  terms  of  computing  speed,  the  amount  and  types  of  data 
required  and  model  sophistication.  These  decrements  represent  the  Hdelity  costs 
inherent  in  the  fidelity  enhancement  process.  Other  than  the  relatively  low  software 
modification  costs,  what  exactly  are  these  fidelity  costs? 
a  Performance  Degradation 

It  is  easy  to  predict  that  the  enhanced  model  will  experience  longer 
execution  times  if  it  is  run  on  the  current  hardware.  But,  the  improved  speed  of 
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newer  hardware  may  compensate  for  the  longer  execution  times.  Therefore,  model 
migration  is  one  option  that  should  be  considered.  A  move  to  a  more  capable  machine 
may  be  necessaiy  to  realize  the  needed  enhancements.  However,  this  move  must  be 
approved  in  advance  by  the  end  user  to  verify  his  ability  to  support  the  new  hardware 
requirements.  The  developer  must  then  test  the  enhanced  model  on  the  proposed 
platform  and  budget  his  expanded  capability  accordingly. 

b.  Model  Sophistication 

To  increase  the  detail  of  a  submodel,  the  developer  increases  the 
required  level  of  understanding  for  himself  and  the  user.  Although  the  user  has  the 
option  of  viewing  the  submodel  as  a  "black  box",  the  acceptance  and  confidence  in  the 
answers  rendered  will  normally  necessitate  the  user’s  comprehension  and 
understanding  of  the  model’s  risk  areas.  His  ability  to  effectively  utilize  an  enhanced 
user  interface  may  also  depend  on  thorough  understanding  of  the  model’s  internal 
processes.  The  developer,  on  the  other  hand,  must  be  an  expert.  His  expertise  should 
encompass  the  physical  system  being  modelled  as  well  as  the  model  itself.  This 
knowledge  is  crucial  to  the  definition  and  application  of  the  appropriate  parameters 
to  the  model.  Any  shortcomings  in  this  area  also  increase  data  risk. 

c.  Data  Risk 

Data  risk  derives  from  the  effects  of  increased  resolution  on  the  data 
required  to  run  the  enhanced  model.  One  effect  is  the  need  for  additional  data  to 
support  the  greater  level  of  detail  being  modelled.  Another  effect  is  increased 
sensitivity  to  the  accuracy  of  the  data.  The  developer  and  user  are  often  called  upon 
to  estimate  the  parameters  or  even  the  probability  distributions  used  by  the  model. 
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The  degree  of  confidence  (or  lack  thereof)  in  the  user’s  ability  to  obtain  the  data  and 
the  quality  of  the  sources  make  up  an  enhancement’s  data  risk. 

Unlike  the  first  two  fidelity  costs,  certain  data  risks  can  be  addressed 
through  sensitivity  analysis.  The  ability  to  quantify  these  risks  or  demonstrate  their 
limited  impact  may  alleviate  their  costs.  This  is  accomplished  by  var3dng  the 
enhancement’s  parameters  to  gauge  the  model’s  response.  It  may  also  be  useful  to 
test  each  enhancement  with  varied  parameters.  Each  variant  would  be  tested  as  a 
separate  enhancement.  These  related  variants  would  then  be  compared  to  determine 
the  sensitivity  of  the  enhancement  to  its  parameters.  This  analysis  would  take  place 
in  conjunction  with  the  determination  of  the  fidelity  benefits. 

2.  Fidelity  BeneHts 

The  fidelity  benefits  from  individual  enhancements  manifest  themselves 
as  incrementally  better  answers  to  the  questions  being  asked  or  choices  being  made. 
This  higher  resolution  level  may  raise  the  end-user’s  confidence  in,  and  acceptance  of, 
the  model’s  decisions.  Depending  on  the  upgrade  involved,  the  model  may  answer 
new,  more  detailed  questions  or  provide  more  detailed  answers  to  existing  questions. 
Another  area  that  may  benefit  from  the  upgrade  is  the  accuracy  of  the  decision 
rendered.  These  possible  effects  are  dependent  on  the  particular  model  and  seem 
difficult  to  quantify.  But  the  ability  to  quantify  the  fidelity  benefit  or  yield  of  each 
enhancement  is  crucial  to  the  Fidelity  Enhancement  Process.  This  quantification  is 
the  focus  of  the  fidelity  assessment. 
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3. 


Fidelity  Assessment 

The  fidelity  assessment  is  the  cornerstone  of  the  Fidelity  Enhancement 
Process.  It  encompasses  the  collection  and  processing  of  all  the  fidelity  costs  and 
benefits.  The  assessment  begins  with  the  establishment  of  the  test  case,  which 
includes  the  selection  of  the  data  sets  necessary  to  run  the  model.  Each  of  these  data 
sets  contain  one  of  the  model’s  alternatives,  which  is  a  possible  solution  that  is 
compared  to  all  of  the  other  possible  solutions  to  determine  the  best.  Once  the  set  of 
possible  alternatives  have  been  identified,  the  next  step  is  to  establish  the  decision 
boundaries. 

The  decision  boundaries  are  established  in  terms  of  the  baseline  and 
topline  cases.  Each  of  these  cases  represent  an  upgrade  combination;  a  combination 
of  enhancements  that  are  turned  "on"  and  "off.  The  baseline  case  corresponds  to  no 
enhancements  turned  "on".  This  case  produces  an  answer  equivalent  to  that  of  the 
original  model.  The  decision  produced  by  switching  all  of  the  enhancements  "on"  is 
the  topline  case.  This  case  represents  the  decision  produced  at  maximum  resolution. 
The  first  question  to  be  answered  is:  are  the  baseline  and  topline  decisions  equal?  If 
they  are,  the  enhancements  have  produced  an  insignificant  increase  in  model  fidelity 
and  should  be  left  out  of  the  model,  avoiding  their  fidelity  costs.  If  the  decisions 
differ,  the  fidelity  analysis  continues  with  the  determination  of  a  weight  for  each 
alternative. 

The  model’s  decision  output  from  the  topline  case  is  utilized  to  construct 
the  weights  used  by  the  Meeisure  of  Performance  (MOP)  equation  (below).  A  weight 
is  determined  for  each  of  the  alternatives  outlined  earlier.  Each  alternative’s  weight 
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corresponds  to  the  proportion  of  time^  it  was  chosen  as  the  best  alternative  by  the 
topline  case. 


j  =  Alternative  number 

Wj  =  Weight  of  Alternative  j 

Tj  =  Number  of  Alternative  j  decisions 

R  =  Number  of  Replications 


The  MOP  can  then  be  calculated  for  each  proposed  upgrade  combination  by  running 
R  replications  of  the  model  and  summing  the  products  of  the  weights  and  the  number 
of  times  each  alternative  wat  chosen. 


MOP,  =  i  E  Wj 

7»1 


i  =  Upgrade  Combination  Number 

j  =  Alternative  Number 

MOP^=  Measure  of  Performance  for  Upgrade  Combination  i 
A^j  =  Number  of  Alternative  j  Decisions 
Wj  =  Weight  of  Alternative  j 
R  =  Number  of  Replications 
n  =  Number  of  Alternatives 


The  evaluation  of  these  MOPs  and  the  direction  along  which  the  fidelity  assessment 
proceeds  become  model  dependent  at  this  point. 

For  an  upgrade  involving  a  relatively  small  number  of  enhancements,  a  2'' 
factorial  design  is  preferred.  This  special  case  of  general  factorial  design  is  keyed  to 
the  comparison  of  factors  with  only  two  levels.  In  this  case,  "k"  represents  the  number 
of  enhancements  while  the  two  levels  are  "on"  and  "off.  This  type  of  analysis  allows 


18 


the  analyst  to  more  accurately  gauge  the  interactions  between  the  k  factors  and  their 
effect  on  the  model’s  response[Ref.  8].  For  upgrades  that  entail  numerous 
enhancements,  the  number  of  experimental  runs  required  for  a  factorial  design  may 
become  prohibitive.  Alternate  designs,  such  as  two-stage  designs  or  single  factor 
analysis  are  more  appropriate  for  these  situations. 

Single  factor  analysis  treats  each  enhancement  as  an  alternative  system  and 
discounts  the  interactions  between  them.  Common  examples  of  this  technique  include 
randomized  complete  block  design  and  other  forms  of  one  way  analysis  of  variance  [Ref. 
[Ref  9].  Or,  as  an  alternative,  the  best  of  a  group  of  similar  enhancements  may 
be  chosen  using  "two-stage"  sampling.  This  technique  also  treats  each  enhancement 
as  an  individual  alternative.  The  analyst  estimates  the  variance  produced  during  the 
first  stage.  This  is  used  to  establish  the  number  of  runs  required  for  the  second  stage, 
which  produces  the  final  decision[Ref  9].  A  slight  modification  of  this  technique 
selects  a  fixed  number  of  best  enhancements  from  the  total[Ref  9].  This  method  can 
be  used  to  trim  down  the  number  of  enhancements  to  a  level  more  conducive  to  a 
factorial  design.  This  brief  look  provides  some  ideas  concerning  different  possibilities 
for  enhancement  analysis.  It  remains  the  job  of  the  analyst  to  tailor  this  assessment 
to  the  characteristics  and  needs  of  his  model  upgrade  and  provide  the  decision  maker 
with  the  data  needed  to  make  an  informed  fidelity  decision. 

F.  STAGE  5  -  FIDELITY  DECISION 

The  fidelity  decision  stage  consolidates  the  analysis  of  fidelity  benefits  with  that 
of  the  fidehty  costs.  The  end  user  then  compares  the  enhancement  yields  with  their 
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associated  performance  decrements  to  the  subjective  analysis  of  model  sophistication 
and  data  risk  and  makes  his  decision. 
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IV.  APPUCATION  OF  THE  PROCESS 


A.  BACKGROUND 

The  Fidelity  Enhancement  Process  was  applied  to  the  Marine  Corps 
Communication  Architectxire  Analysis  Model  (MCCAAM).  This  proved  beneficial  even 
though  MCCAAM  was  in  its  initial  development. 

MCCAAM  is  a  computer  simulation  of  Marine  Corps  single-channel  radio 
architectures[Ref.  4].  It  replicates  the  interactions  between  units,  radios  and  nets  in 
a  realistic  manner  using  the  BOST  message  structure  explained  in  section  n  D.  The 
network  architecture  is  constructed  dynamically  from  the  input  data,  which  results  in 
nearly  unlimited  flexibility:  the  model  can  be  applied  to  radio  networks  of  all  t5T)es  and 
sizes.  A  penalty  process  gauges  the  number  of  BOSTs  that  are  not  completed  in  a 
timely  maimer.  Each  BOST  has  an  allotted  time  for  completion  after  which  the 
architecture  is  immediately  assessed  a  BOST-specific  one  time  penalty  and  then  a 
(again  BOST-specific)  constant  penalty  rate  until  it  is  completed.  This  penalty  process 
is  used  to  assess  the  performance  of  a  given  architecture  in  terms  of  its  long  term 
penalty  rate. 

The  original  (baseline)  model  uses  the  internal  penalty  process  to  choose  the  best 
architecture  from  a  finite  set  of  alternatives.  In  addition  to  this  capability,  the 
problem  of  choosing  the  best  allocation  of  new  SINCGARS  radios  as  partied 
replacements  for  current  PRC-77  radios  in  an  existing  architecture  was  posed.  This 
particular  application  of  MCCAAM  required  enhancements  that  would  differentiate 
between  the  two  radio  tjqies. 
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B.  STAGE  1 .  THE  MODEL  ASSESSMENT 


The  model  assessment  proceeded  rapidly  because  the  upgrade  was  executed 
almost  concurrently  with  the  model’s  initial  development.  As  a  result,  the  risk  areas 
were  up  to  date,  the  model’s  foundation  was  sound  and  the  boundaries  were 
estabhshed  during  the  original  development  effort. 

The  model’s  hardware  boundaries  were  dictated  by  the  software  memory 
limitations  of  MODSIM’s  PC  version  and  its  C  compiler.  The  model  quickly  outgrew 
that  platform  and  was  subsequently  migrated  to  a  SUN  workstation  with  a  newer 
version  of  MODSIM II.  The  previously  encountered  limitations  were  alleviated  by  this 
move,  which  was  cleared  by  the  end-user  prior  to  its  adoption  by  the  developers. 

C.  STAGE  2  -  FroEUTY  ENHANCEMENT  REQUHIEMENTS 

This  model-driven  upgrade  resulted  in  the  development  of  numerous 
enhancements  including  a  new  object,  which  portrayed  the  effects  of  enemy  jamming 
systems,  and  changes  to  existing  objects. 

1.  Jammer  Object 

The  introduction  of  enemy  jamming  to  the  model  was  considered  crucial 
because  the  newer  SINCGARS  radios  have  a  frequency  hopping  capability  that  make 
them  effectively  "jam-proof,  and  this  is  the  primary  difference  between  the  two  radios. 
The  new  Jammer  object  type  was  developed  as  a  generic  specification  that  could  be 
apphed  to  any  enemy  jamming  system.  The  parameters  used  to  specify  each  jai-uner 
include  its  location  and  type  as  well  as  the  jamming  direction,  range  and  duration. 
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2.  Mean  Time  Between  Failures 


An  additional  discriminator  between  the  two  radios  is  the  expected  mean 
time  between  failures  (MTBF).  Test  and  evaluation  of  the  new  radio  reflects  a 
significant  increase  in  reliability  for  the  SINCGARS.  In  addition,  the  modular  design 
of  the  SINCGARS  radio  gives  it  a  shorter  mean  repair  time.  However,  once  the 
SINCGARS  radio  is  repaired  or  replaced,  the  process  of  rejoining  a  frequency-hopping 
SINCGARS  net  requires  significantly  more  time  than  rejoining  a  PRC-77  single 
frequency  net.  As  a  result,  parameters  reflecting  the  MTBF,  repair  time,  and  net 
entry  time  for  each  radio  type  were  estimated.  The  radio  object  and  its  methods 
required  modification  to  incorporate  these  changes. 

3.  Perishability 

The  developers  found  that  the  need  for  action  following  the  loss  of  a  radio 
or  access  to  a  net  induced  a  complicated  series  of  events.  Although  the  routing  of 
traffic  on  a  functional  network  is  relatively  straight  forward,  the  alternate  routing 
procedures  required  to  deal  with  enemy  jamming  and  equipment  failure  proved  to  be 
very  complex.  Incorporating  an  algorithm  to  accomplish  the  alternate  routing  required 
the  addition  of  a  completely  new  modcde. 

Bottlenecks  formed  in  the  network  by  the  previously  mentioned  changes  raised 
questions  concerning  the  perishability  of  message  traffic.  A  message  that  is  trapped 
in  an  inoperational  radio’s  queue  may  reach  a  point  where  its  transmission  is  no  longer 
a  valid  requirement.  At  this  point,  the  message  is  considered  perishable  and  is 
removed  from  the  network.  The  enhancements  required  to  implement  this  capability 
included  the  modification  of  the  BOST  structure  and  changes  to  the  BOST  data  files. 
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D.  STAGE  3  -  PROTOTYPING 


The  enhancements  described  above  were  systematically  added  during  model 
development.  The  software  switches  used  to  enable  the  enhancements  could  be 
switched  on  to  work  on  the  enhancements  or  off  for  unimpeded  work  on  the  baseline 
model.  By  adhering  to  this  practice,  the  development  of  both  proceeded  with  minimal 
conflict. 

E.  STAGE  4  -  FIDELITY  ANALYSIS 

Three  enhancements  were  implemented  to  differentiate  between  SINCGARS 
radio  allocations:  Jammers,  MTBF,  and  Perishability. 

1.  Fidelity  Costs 

The  fidelity  costs  incurred  by  the  chosen  enhancements  include 
performance  degradation,  model  sophistication  and  data  risk.  Performance  degradation 
was  measured  in  terms  of  clock  time.  A  stopwatch  was  used  to  measure  the  extra  time 
required  for  each  of  the  enhancements.  In  addition,  the  subjective  costs  were 
evaluated  for  each  of  the  enhancements, 
a.  Jammer  Object 

The  addition  of  the  jammer  object  introduced  a  great  deal  of  model 
sophistication  and  data  risk.  The  baseline  model  and  its  portrayal  of  the  MAGTF  radio 
network  is  comprehendible  to  anyone  with  experience  in  tactical  operations.  However, 
the  proper  appUcation  of  electronic  warfare  to  the  model  requires  additional  expertise 
on  the  part  of  the  user.  The  choice  of  jammer  type  and  employment  strategy  must  be 
made  during  the  creation  of  the  jammer  data  file  and  prior  to  model  execution.  Data 
risk  is  generated  by  the  choices  previously  mentioned  as  well  as  the  jammer’s 
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parameters  which,  in  this  case,  do  not  reflect  terrain  features  or  weather 
considerations. 

b.  Radio  Failures 

Although  the  sophistication  required  to  utilize  the  radio  failure 
enhancement  is  comparable  to  that  required  by  the  baseline  model,  there  is  some  data 
risk  involved.  The  actual  MTBF  is  an  estimated  parameter  as  is  the  "repair  or  replace" 
time  for  each.  Other  data  risk  issues  involve  modelling  the  substitution  of  broken 
radios  with  spares  or  switching  frequencies  between  nets  on  an  operational  radio. 

c.  Message  Perishability 

Message  perishability  is  primarily  data  risk  sensitive.  These  risks 
include  a  subjective  judgement  whether  or  not  each  BOST  is  perishable  as  well  as  the 
estimation  of  perishability  points  for  those  that  are. 

2.  Fidelity  Benefits 

The  fidelity  benefits  manifest  themselves  as  better  radio  allocation 

decisions. 

3.  Fidelity  Assessment 

a.  Selection  of  Alternatives 

The  fidelity  assessment  began  with  the  specification  of  the  allocation 
alternatives.  A  sample  data  set  was  developed  that  focused  on  the  Ground  Combat 
Element  (GCE)  of  a  Marine  Expeditionary  Brigade  (MEB).  The  vmits,  nets  and  BOSTs 
that  were  stressed  involved  indirect  fire  support  and  tactical  communications.  Three 
different  radio  allocations  were  chosen  as  possible  solutions  to  the  optimal  allocation 
problem.  These  three  radio  allocations  served  as  the  alternatives. 
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Table  I  Allocation  Alternatives 


1  ALTERNATIVES  | 

Number 

Description 

#  SINCGARS 

1 

FEBA  Back  Configuration 

49 

2 

Top  Down  Configuration 

53 

3 

All  PRC-77  Configuration 

0 

The  first  two  alternatives  reflect  the  possible  tactical  employment  of 
these  communications  assets  on  the  battlefield  while  the  third  represents  the  current 
architecture.  The  forward  edge  of  the  battle  area  (FEBA)  back  alternative  places  the 
SINCGARS  radios  on  those  nets  that  are  physically  closest  to  enemy  jamming  assets 
and  carry  the  bulk  of  the  architecture’s  traffic  load.  These  include  nets  of  battalion 
level  and  lower.  The  top  down  alternative  focuses  the  emplo5maent  of  SINCGARS 
radios  in  the  nets  controlled  by  higher  headquarters.  These  nets  normally  have  a 
greater  number  of  subscribers  and  process  the  most  important  message  traffic.  The 
third  alternative  depicts  the  current  tactical  architecture  with  no  SINCGARS  radios 
employed.  This  alternative  was  added  to  judge  the  impact  of  no  SINCGARS  radios  on 
the  scenario  and  provide  a  measure  of  current  architecture  performance. 
b.  Ejq)erimental  Design 

A  2'‘  factorial  experimental  design  was  selected.  The  name  2*'  relates 
to  considering  k  factors  each  with  two  possible  levels.  This  design  allows  the  smallest 
number  of  treatment  combinations  with  which  k  factors  can  be  analyzed  under  a 
complete  factorial  arrangementIRef  8]. 
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A  2*^  design  may  provide  information  on  how  sensitive  the  model’s 
output  is  to  the  different  enhancements.  This  design  can  also  provide  insight  into  the 
interactions  between  the  enhancements.  The  design  is  particularly  well  suited  to  this 
fidelity  analysis  in  that  the  enhancements  are  inherently  two  level  (on  and  off). 

The  2^  design  represents  an  experiment  using  three  factors  each  with 
two  levels.  As  stated  earlier,  the  three  factors  analyzed  were  jammers,  MTBF,  and 
perishability.  The  incorporation  of  these  three  factors  produced  8  possible  treatment 
(upgrade)  combinations. 


Table  11  Treatment  (Upgrade)  Combinations 


Upgrade  Combinations  || 

index 

code 

description 

1 

000 

Baseline  Case  (all  ofD 

2 

001 

Jammers  only 

3 

010 

MTBF  only 

4 

on 

Jammers  &  MTBF 

5 

100 

Perishability  only 

6 

101 

Jammers  &  Perishability 

7 

no 

MTBF  &  Perishability 

8 

111 

Topline  Case  (all  on) 

The  upgrade  combinations  (UC)  are  indexed  with  i  which  ranges  from  1  to  8.  The 
code  represents  the  actual  enhancements  incorporated  within  each  upgrade 
combination.  The  three  digits  of  the  code  correspond  to  the  three  enhancements 
portrayed  as  perishability,  MTBF  and  Jammers  in  that  order.  The  digits  0  and  1 
correspond  to  off  and  on,  respectively. 
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Each  of  the  radio  allocation  alternatives  were  evaluated  by  MCCAAM 
under  each  of  the  8  upgrade  combinations.  Each  of  these  24  (3  alternative  x  8  upgrade 
combinations)  model  runs  produced  a  steady  state  penalty  plot  that  was  analyzed  using 
MCCAAM’s  analysis  routines.  This  analysis  was  performed  on  the  penalty  rates  (Ry^) 
generated  by  each  model  run.  The  initial  condition  analysis  established  the  steady 
state  point  at  700  minutes  into  the  10,000  minute  rxm.  The  remaining  9300  minutes 
were  sampled  at  25  minute  intervals  to  produce  approximately  360  samples.  The  batch 
size  was  set  at  12  to  produce  30  iid  batch  means  or  penalty  rates.  The  autocorrelation 
of  these  batches  was  analyzed  and  found  to  be  insignificant(max  p  =  .023120).  Each 
vector  of  penalty  rates  was  then  broken  down  into  five  groups  of  six  samples  and 
tabulated  in  matrices. 


Table  in  Sample  Ryi^  Matrix 


UC  2 

Samples  (k  =  l,... ,6)  for  Re 

jlication  1 

Alternative 

1 

2 

3 

4 

5 

6 

1 

mm 

mm 

l3Mi 

mm 

warn 

^216 

2 

mm 

I^HH 

3 

^231 

R232 

^33 

^4 

^236 

J^236 

=  penalty  rate  k  of  UCi  and  alternative j  ,  Jc  =  l,  ,  .  .  ,30 

These  penalty  rates  are  then  compared  across  the  alternatives  to 
determine  the  lowest  penalty  rate  for  each  sample  number.  The  variable  Ty^  is  used 
to  denote  the  alternative  with  the  lowest  (best)  penalty  rate.  The  winner  receives  a 
one  while  the  remainder  of  the  alternatives  get  zeros. 
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^ijJc  =  1'  ^11  n*j 

=  0,  Otherwise 


These  values  were  then  tabulated  in  the  table  and  the  group  totals  were  calculated 

by  summing  across  the  samples. 

Table  IV  Sample  Tj,k  Matrix 


UC2 

Samples 

Group 

Totals 

Alternative 

■■ 

2 

3 

4 

5 

6 

1 

0 

0 

0 

1 

0 

1 

2 

2 

1 

0 

mm 

0 

1 

0 

3 

3 

0 

1 

0 

0 

0 

0 

1 

These  group  totals  (A^j,)  reflect  the  number  of  times  each  alternative 
was  selected  as  the  best  and  range  from  zero  to  six.  The  group  totals  are  used  to 
calculate  the  Measure  of  Performance  (MOP)  values  for  each  group.  Once  calculated, 
the  group  totals  are  tabulated  in  a  group  total  matrix. 

i(6) 

A^ji  =  ,  1  =  1,  ...  ,5 


i  =  Upgrade  Combination  (C7C)  index 

j  =  Al  terna  ti  ve  Number 

k  =  Sample  Number 

1  =  Group  Number 

A^j2  =  Number  of  Times  Alternative  j  chosen  for  UC  i  in  Group  1 
=  Lowest  Penalty  Rate  for  Sample  k,  lorO 
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Table  V  Sample  Group  Total  Matrix 


The  measure  of  performance  (MOP)  for  each  upgrade  combination  is 
calculated  by  summing  the  products  of  weight  and  number  of  times  chosen  for  each 
alternative.  This  sum  is  then  divided  by  the  group  sample  size  to  get  the  average 
value  for  each  group.  These  40  MOP  values  are  then  used  to  conduct  an  analysis  of 
variance  ANOVA. 

3 

MOP^^  =  i  E  ^  =  1' - ®  '  -Z  =  l/  •  •  -  5 

i-i 

i  =  Upgrade  Combination  (UC)  index 

j  =  Alternative  Number 

1  =  Group  Number 

MOP^j  =  Measure  of  Performance  for  UC  i 

=  Number  of  Times  Alternative  j  chosen  for  UC  i 
Wj  =  Weight  of  Alternativej 

The  weight  for  each  alternative,  Wj  was  calculated  using  the  topline  case  as  outlined 
in  section  III  E  3.  This  weight  represents  the  proportion  of  the  time  that  an 
alternative  was  selected  by  the  highest  fidelity  upgrade  combination  (the  topline  case). 
The  group  totals  were  sximmed  for  each  alternative  and  then  divided  by  the  total 
number  of  samples  (30)  to  determine  each  alternative’s  weight. 


Table  VI  Treatment  (Upgrade)  Combinations 


1  Upgrade  Combinations 

index 

code 

description 

MOP 

1 

000 

Baseline  Case  (all  off) 

(1) 

2 

001 

Jammers  only 

a 

3 

010 

MTBF  only 

b 

4 

Oil 

Jammers  &  MTBF 

ab 

5 

100 

Perishability  only 

c 

6 

101 

Jammers  &  Perishability 

ac 

7 

110 

MTBF  &  Perishability 

be 

8 

111 

Topline  Case  (all  on) 

abc 

Once  all  of  the  MOP  values  have  been  determined,  multifactor  analysis 
of  variance  (ANOVA)  was  utilized  to  calculate  the  relationship  between  the  response 
variable  (fidelity  jdeld)  and  the  three  factors.  The  model’s  accuracy  depends  on  the 
assumption  that  the  error  terms  are  normally  distributed  and  independent.  The 
following  linear  equation  was  used  to  model  this  relationship. 

th  response  with  factors  at  levels  i,  j,  k 
p  =  overall  mean  response 
T_i  =  effect  of  Jammers  at  level  i 
Pj  =  effect  of  MTBF  at  level  j 
Y*  =  effect  of  Perishability  at  level  k 
the  random  error  component 
ijk=  factor  level  {l=on.  0  =off) 

1  =  sampl  e  number  {1-1,  .  .  . ,  6 ) 
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In  this  design,  the  factors  tj,  Pj,  and  Yk  correspond  to  the  three 
enhancements.  The  terms  in  parentheses  indicate  two-way  (t p )y  and  three-way  (t p Y)yk 
interactions  of  the  corresponding  factors.  It  is  assumed  that  the  random  error  terms, 
:^-u,  are  independent,  identically  distributed  normal  variables  with  a  mean  of  zero  and 
a  variance  of  o^[Ref.  8].  The  treatment  effects  are  defined  as  deviations  from  the 
overall  mean  so, 

=  =  EYjc  =  0 

i*l  j-l 


Similarly,  the  interaction  effects  are  fixed  and  defmed  so  that  they  also  sum  to  zero 
as  shown  below  for  the  (tP)^j  interaction. 

t  -  t  '  0 

1-1  J-l 


In  this  factorial  design,  all  three  factors  are  of  equal  interest.  We  are  specifically 
interested  in  testing  hypotheses  concerning  the  equality  of  the  treatment  effects. 

"^i  =  ■'2  =  0 

at  least  one  *  0 

We  are  also  interested  in  the  interactions  between  the  treatments  and  therefore  test 
each  of  the  interaction  terms. 

(xP)^^  =  0  for  all  i,j 
at  least  one  (xP)^j-  *  0 
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These  hypotheses  are  tested  using  a  multifactor  ANOVA.  The  sum  of 
squares  for  each  treatment  as  well  as  the  total  sum  of  squares  (SS)  is  calculated  and 
divided  by  its  degrees  of  freedom  to  obtain  its  mean  square.  The  expected  values  of 
the  mean  squares  (MS)  are 

bn  ^  Ti 

E{MS.)  =  E(^)  =  0^  + - — 

a-i  3-1 

a=b=  2  levels 
n=  5  groups  of  data 


If  the  null  hypothesis  for  each  treatment  is  true,  then  all  of  the  expected  mean 
squares  estimate  o^.  However,  if  there  are  differences  between  treatment  effects  then 
that  particular  E(MS)  value  will  be  larger  than  the  expected  mean  square  error  term 
E(MSe). 

EiMS^)  = 

Therefore,  to  test  the  significance  of  the  main  effects  and  their  interactions,  simply 
divide  the  corresponding  mean  square  by  the  mean  square  error.  Based  on  our 
assumption  of  independent  identically  distributed  normal  error  terms  with  constant 
variance,  o^,  each  of  the  ratios  of  mean  squares  are  distributed  as  F  with  1  degree  of 
freedom  in  the  numerator  and  32  in  the  denominator.  The  critical  region  is  then  the 
upper  tail  of  the  F  distribution.  The  procedure  is  summarized  in  the  analysis  of 
variance  table  in  part  d.  of  this  subsection. 
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c.  Weight  DetermiTiation 


The  weights  were  calculated  by  using  the  topline  case  to  generate 
decision  output  that  reflects  the  highest  level  of  fidelity.  This  data  was  transformed 
into  proportions  that  reflect  the  frequency  that  the  alternative  was  chosen. 


Table  Vn  Alternative  Weights 


Alternative 

#  Selections 

Weight 

1 

18 

.60 

2 

5 

.17 

3 

7 

.23 

d.  Results  of  the  2?  Experiment 

The  experiment  was  performed  as  discussed  in  the  part  b.  The  penalty 


rate  (Ry^)  and  Ty,j  matrices  are  consohdated  in  Appendix  A.  The  Group  Total  matrices 
are  also  located  in  Appendix  A.  The  MOP  values  (MOP;,)  are  displayed  below. 

Table  VIII  MOP  Values 


Groups 

UC 

1 

2 

3 

4 

5 

1 

.333 

.333 

.333 

.333 

.333 

2 

.323 

.395 

.343 

.467 

.405 

3 

.262 

.343 

.395 

.405 

.343 

4 

.272 

.415 

.405 

.467 

.282 

5 

.333 

.333 

.333 

.333 

.333 

6 

.323 

.395 

.343 

.467 

.405 

7 

.415 

.467 

.252 

.405 

.343 

8 

.395 

.467 

.395 

.538 

.415 
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These  values  were  then  analyzed  using  the  multifactor  ANOVA 


capabilities  of  Statgraphics  version  5.  The  resulting  ANOVA  table  revealed  only  one 
significant  main  effect  and  no  significant  interactions. 

Table  IX  ANOVA  Results 


1  Analysis  of  Variance  for  Fidelity  Yield  { 

Source  of 

Variation 

Sum  of 
Squares 

df 

Mean 

Square 

F- 

ratio 

P 

Main  Effects 

A 

.81796 

1 

.81796 

6.579 

.015 

B 

.20736 

1 

.20736 

1.668 

.206 

C 

.22801 

1 

.22801 

1.834 

.185 

Interactions 

AB 

.01156 

1 

.01156 

.093 

.766 

AC 

.05041 

1 

.05041 

.405 

.536 

BC 

.22801 

1 

.22801 

1.834 

.185 

ABC 

.05041 

1 

.05041 

.405 

.536 

Residual 

3.97872 

32 

.124335 

Toted 

5.57244 
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The  application  of  Jammers  produced  the  only  significant  effect  on  the  model’s  fidelity 
yield(p  =  .015).  The  remaining  two  factors,  MTBF  and  perishability  produced  roughly 
equivalent  p  values  (.206  and  .185  respectively)  but  were  well  above  a  conservative  a 
value  of  .10.  The  interaction  effects  were  even  less  significant  with  the  exception  of 
the  BC  term  which  corresponds  to  the  interaction  between  MTBF  and  perishability. 
This  term  had  a  p  value  of  .185  while  the  remainder  were  greater  than  0.5.  The 
interaction  plots  are  in  Appendix  B. 
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F.  STAGE  5  -  FIDELITY  DECISION 


The  fidelity  analysis  demonstrates  clearly  that  only  the  Jammer  enhancement 
should  be  added  to  the  upgraded  model.  Its  significance  to  the  model  is  highlighted 
by  the  results  of  the  multifactor  ANOVA  as  displayed  in  Table  IX.  The  fideUty  costs 
of  the  remaining  enhancements  greatly  outweigh  their  impact  on  the  model’s  decision 
and  should  be  omitted  from  the  SINCGARS  allocation  determination  process.  A 
comprehensive  record  of  these  enhancements  should  be  maintained  however,  because 
they  may  prove  beneficial  to  a  later  upgrade  effort  or  their  present  fidehty  costs  may 
be  reduced  by  some  new  data  source. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Computer  simulation  is  poised  on  the  threshold  of  an  exciting  new  frontier.  The 
continuing  technological  advances  in  computing  power,  both  in  hardware  and  software, 
have  provided  a  software  development  environment  that  is  conducive  to  model  reuse. 
Object-Oriented  simulation  and  open  architecture  systems  are  primary  examples  of 
advances  that  impact  directly  on  model  reuse.  The  diminishing  requirements  for 
model  reimplementation  will  increase  the  availability  of  modelling  effort  for  both  new 
model  development  and  the  improvement  of  existing  models.  To  properly  navigate 
within  this  new  frontier,  a  specialized  methodology  is  required. 

The  Fidelity  Enhancement  Process  provides  a  useful  map  for  conducting  the 
upgrade  of  an  existing  model.  Although  the  process  is  directed  toward  models 
incorporating  OOS  and  open  architecture,  its  five  stages  address  all  of  the  steps 
necessary  for  a  successful  model  upgrade.  The  individual  stages  are  general  enough 
to  allow  their  apphcation  to  a  wide  range  of  decision  making  models.  This 
methodology  was  tested  during  the  upgrade  of  the  Marine  Corps  Communication 
Architecture  Analysis  Model(MCCAAM). 

The  Fidelity  Enhancement  Process  received  its  initial  apphcation  during  the 
upgrade  of  MCCAAM.  The  stages  proved  beneficial  in  structuring  the  upgrade  process 
to  allow  the  rapid  apphcation  of  the  required  enhancements.  The  prototyping  was  very 
conducive  to  a  group  development  effort  in  that  the  enhancements  could  be  turned  off 
to  negate  their  impact  on  the  remainder  of  the  model.  The  fidelity  analysis  allowed 
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the  developers  to  fine  tune  the  final  model  and  limit  the  fidelity  costs.  The  2'‘  factoral 
design  proved  to  be  an  effective  technique  for  assessing  the  interactions  between  the 
enhancements  as  well  as  the  main  effects.  The  2*'  factoral  experimental  design 
provides  a  solid  foundation  for  the  fidelity  analysis  stage.  The  key  to  the  continued 
usefulness  of  the  Fidelity  Enhancement  Process  is  the  expansion  of  this  stage  by 
increasing  the  number  of  different  analysis  techniques  used. 

B.  RECOMMENDATIONS 

The  tremendous  potential  of  model  reuse  warrants  continued  emphasis.  The 
Fidelity  Enhancement  Process  should  be  applied  to  more  models  to  validate  its  stages 
and  expand  the  number  of  documented  fidelity  analysis  techniques.  A  greater  variety 
of  well-documented  analysis  strategies  and  techniques  will  increase  the  usefulness  of 
the  process  by  providing  more  analysis  options  for  its  users.  The  additional 
applications  may  also  uncover  the  need  for  modifications  to  the  stages. 

The  fidelity  analysis  stage  presents  the  most  potential  for  expansion  or 
modification.  Fidelity  analysis  is  currently  associated  with  the  overall  effect  of  an 
enhancement  being  turned  on  or  off.  Additional  insight  may  be  gained  by  conducting 
sensitivity  analysis  on  an  enhancement.  This  may  alter  the  fidelity  analysis  or  become 
part  of  the  presentation  of  results.  The  actual  incorporation  of  sensitivity  analysis  into 
the  fidelity  analysis  is  model  dependent  at  this  point,  but  any  techniques  utiUzed  to 
address  this  issue  will  aid  future  users  in  tackling  these  problems. 
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APPENDIX  A:  DATA 


Upgrade  Combination  (1) 

code  000 

Alternative 

Grand  Mean 

Standard 

Deviation 

Auto 

Correlation 

1 

4.44 

1.66 

.001869 

2 

4.44 

1.66 

.001869 

3 

4.44 

1.66 

.001869 

Sam] 

lies 

Alternative 

1 

2 

3 

4 

5 

6 

1 

5.92 

6.38 

7.43 

2.44 

6.01 

2.80 

2 

5.92 

6.38 

7.43 

2.44 

6.01 

2.80 

3 

5.92 

6.38 

7.43 

2.44 

6.01 

2.80 

Sam] 

ales 

Alternative 

7 

8 

9 

10 

11 

12 

1 

5.52 

4.77 

3.63 

3.15 

1.69 

2.55 

2 

5.52 

4.77 

3.63 

3.15 

1.69 

2.55 

5.52 

4.77 

3.63 

3.15 

1.69 

2.55 

Upgrade  Combination  1 


code  000 


Samples 

13 

14 

15 

16 

17 

18 

3.10 

2.83 

4.57 

6.90 

6.52 

1.76 

3.10 

2.83 

4.57 

6.90 

6.52 

1.76 

3.10 

2.83 

4.57 

6.90 

6.52 

1.76 

Alternative 


Alternative 


Samples 


19 

20 

21 

22 

23 

24 

2.88 

5.12 

5.64 

4.14 

2.38 

5.94 

2.88 

5.12 

5.64 

4.14 

2.38 

5.94 

2.88 

5.12 

5.64 

4.14 

2.38 

5.94 

Samples 

25 

26 

27 

28 

29 

30 

3.48 

3.08 

5.57 

5.85 

6.77 

4.51 

3.48 

3.08 

5.57 

5.85 

6.77 

4.51 

3.48 

3.08 

5.57 

5.85 

6.77 

4.51 

Alternative 


Tyk  TABLES 


UC  1 

Samples 

Group 

Totals 

Alternative 

B 

2 

3 

4 

5 

6 

1 

.33 

.33 

.33 

.33 

.33 

.33 

2 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

UC  1 

Samples 

Group 

Totals 

Alternative 

B 

8 

9 

10 

11 

12 

1 

.33 

.33 

.33 

.33 

.33 

.33 

2 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

3 

.33 

.33 

.33 

.33 

.33 

.33 

2 

1 

Samples 

Group 

Totals 

Alternative 

13 

14 

15 

16 

17 

18 

1 

.33 

.33 

.33 

.33 

.33 

.33 

2 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

3 

.33 

.33 

.33 

.33 

.33 

.33 

2 

UC  1 

Samples 

Group 

Totals 

Alternative 

19 

20 

21 

22 

23 

24 

1 

.33 

.33 

.33 

.33 

.33 

.33 

2 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

3 

.33 

.33 

.33 

.33 

.33 

.33 

2 

UC  1 

Samples 

Group 

Alternative 

25 

26 

27 

28 

29 

30 

Totals 

1 

.33 

.33 

.33 

.33 

.33 

.33 

2 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

3 

.33 

.33 

.33 

.33 

.33 

.33 

2 

Ay,  TABLE 


UC  1 

Alternatives 

Group 

1 

2 

3 

Total 

1 

2 

2 

2 

6 

2 

2 

2 

2 

6 

3 

2 

2 

2 

6 

4 

2 

2 

2 

6 

5 


2 


2 


2 


6 


UPGRADE  COMBINATION  2  RAW  DATA 


Upgrade  Combination  2 


code  001 


Grand  Mean 

Standard 

Deviation 

Auto 

Correlation 

4.49 

1.70 

.010043 

4.83 

1.64 

.018200 

4.59 

1.64 

.013120 

Alternative 


Samples 


Alternative 


Samples 

7 

8 

9 

10 

11 

12 

4.59 

4.91 

3.54 

2.94 

1.56 

2.69 

5.85 

5.62 

4.84 

2.56 

1.69 

2.54 

4.86 

6.70 

4.68 

2.74 

1.12 

3.66 

Upgrade  Combination  2 


code  001 


Alternative 


Samples 

13 

14 

15 

16 

17 

18 

3.08 

3.28 

4.47 

7.04 

5.98 

1.76 

3.10 

3.29 

4.68 

7.01 

6.44 

1.64 

2.52 

3.49 

5.02 

5.70 

5.44 

2.04 

Alternative 


Samples 


19 

20 

21 

22 

23 

24 

2.70 

5.46 

6.04 

5.01 

2.36 

5.74 

3.55 

5.12 

5.86 

5.46 

2.38 

6.20 

3.87 

5.42 

4.77 

5.97 

3.12 

6.70 

Samples 

25 

26 

27 

28 

29 

30 

3.24 

2.89 

5.89 

6.73 

6.79 

4.13 

4.56 

3.42 

6.00 

6.31 

6.83 

5.55 

3.30 

3.37 

2.91 

6.73 

6.79 

4.13 

Alternative 


Tyk  TABLES 


UC2 

Samples 

Group 

Totals 

Alternative 

25 

26 

27 

28 

29 

30 

1 

n 

1 

0 

.5 

.5 

3 

2 

0 

0 

H 

1 

0 

0 

1 

3 

0 

0 

1 

0 

.5 

.5 

2 

Ay,  TABLE 


UC  2 

Alternatives 

Group 

1 

2 

3 

Total 

1 

2 

3 

1 

6 

2 

3 

2 

1 

6 

3 

2 

1 

3 

6 

4 

4 

1 

1 

6 

5 

3 

1 

2 

6 

47 


UPGRADE  COMBINATION  3  RAW  DATA 


Upgrade  Combination  3 


code  010 


Grand  Mean 

Standard 

Deviation 

Auto 

Correlation 

5.02 

1.53 

.011086 

5.18 

1.62 

.002872 

5.14 

1.71 

.015003 

Alternative 


Samples 


1 

2 

3 

4 

5 

6 

6.68 

6.84 

8.08 

4.49 

6.64 

3.84 

6.61 

7.80 

7.91 

4.90 

5.96 

3.57 

6.98 

7.79 

8.32 

2.26 

6.40 

3.50 

Alternative 


Samples 


Upgrade  Combination  3 


code  010 


Samples 

13 

14 

15 

16 

17 

18 

3.83 

2.86 

3.58 

7.02 

5.67 

2.37 

3.67 

3.16 

5.54 

7.00 

6.66 

1.89 

4.59 

3.81 

6.00 

6.76 

6.82 

2.25 

Alternative 


Alternative 


Samples 


19 

20 

21 

22 

23 

24 

3.85 

5.11 

5.93 

5.78 

2.71 

6.08 

4.18 

5.67 

5.73 

4.98 

2.85 

7.04 

5.90 

5.97 

5.40 

6.33 

2.40 

7.20 

Samples 

25 

26 

27 

28 

29 

30 

4.24 

4.87 

6.35 

5.57 

7.39 

4.70 

4.30 

3.88 

5.00 

6.92 

7.18 

5.29 

4.68 

3.38 

5.53 

5.89 

7.08 

4.49 

Alternative 


49 


TABLES 


UC3 

Samples 

Group 

Totals 

Alternative 

1 

2 

3 

4 

5 

6 

1 

0 

1 

0 

0 

0 

0 

1 

2 

B 

0 

B 

0 

1 

0 

3 

3 

0 

0 

0 

1 

0 

B 

2 

UC  3 

Samples 

Group 

Totals 

Alternative 

B 

8 

9 

10 

11 

12 

1 

0 

0 

0 

1 

0 

B 

2 

2 

0 

0 

0 

0 

1 

0 

1 

3 

1 

1 

B 

0 

0 

0 

3 

UC3 

Samples 

Group 

Totals 

Alternative 

13 

14 

15 

16 

17 

18 

1 

0 

1 

B 

0 

1 

0 

3 

2 

B 

0 

0 

0 

0 

B 

2 

3 

0 

0 

0 

1 

0 

0 

1 

UC3 

Samples 

Group 

Totals 

Alternative 

19 

20 

21 

22 

23 

24 

1 

1 

1 

0 

0 

0 

B 

3 

2 

0 

0 

0 

1 

0 

0 

1 

3 

0 

0 

B 

0 

1 

0 

2 

UC3 

Samples 

Group 

Totals 

Alternative 

25 

26 

27 

28 

29 

30 

1 

a 

0 

0 

1 

0 

0 

2 

2 

0 

0 

1 

0 

0 

0 

1 

3 

0 

1 

0 

0 

1 

B 

3 

Ay,  TABLE 


UC3 

Alternatives 

Group 

1 

2 

3 

Total 

1 

1 

3 

2 

6 

2 

2 

1 

3 

6 

3 

3 

2 

1 

6 

4 

3 

1 

2 

6 

5 

2 

1 

- 1 

3 

6 

51 


UPGRADE  COMBINATION  4  RAW  DATA 


52 


Upgrade  Combination  4 


code  Oil 


Alternative 


Samples 

13 

14 

15 

16 

17 

18 

3.55 

3.49 

4.99 

7.16 

5.32 

2.37 

3.71 

4.89 

6.12 

6.98 

7.53 

2.16 

4.81 

4.01 

4.64 

6.60 

7.12 

2.25 

Alternative 


Samples 

19 

20 

21 

22 

23 

24 

3.85 

5.32 

6.17 

5.93 

2.70 

6.09 

4.35 

5.67 

5.42 

6.12 

2.85 

6.39 

4.64 

5.94 

6.02 

6.85 

2.68 

7.37 

Alternative 


Samples 

25 

26 

27 

28 

29 

30 

4.17 

4.08 

6.69 

6.35 

6.13 

4.29 

3 


4.07 

4.19 


4.07 


TABLES 


UC4 


Alternative 


Samples 


2 

3 

4 

0 

0 

0 

0 

0 

0 

5  6 


0 


Group 

Totals 


0 

to 

0 

3 

UC4 


Alternative 


Samples 


2 

0 

0 

CO 

0 

0 

9 

10 

11 

0 

0 

0 

0 

0 

0 

Group 

Totals 


UC4 


Alternative 


Samples 


0 

0 

0 

0 

15 

16 

0 

0 

0 

0 

17  18 


0 


0 


0  0 


Group 

Totals 


Samples 

19 

20 

21 

22 

23 

24 

IB 

1 

0 

1 

0 

n 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

1 

0 

UC4 


Alternative 


Group 

Totals 


UC4 

Samples 

Group 

Totals 

Alternative 

25 

26 

27 

28 

29 

30 

1 

0 

0 

0 

1 

0 

0 

1 

2 

0 

1 

0 

0 

0 

0 

1 

3 

B 

0 

B 

0 

1 

B 

4 

Ay,  TABLE 


UC4 

Alternatives 

Group 

1 

2 

3 

Total 

1 

1 

2 

3 

6 

2 

3 

0 

3 

6 

3 

3 

1 

2 

6 

4 

4 

1 

1 

6 

5 

1 

1 

4 

6 

55 


UPGRADE  COMBINATION  6  RAW  DATA 


Upgrade  Combination  5  code  100 

Alternative 

Grand  Mean 

Standard 

Deviation 

Auto 

Correlation 

1 

4.15 

1.58 

.004884 

2 

4.15 

1.58 

.004884 

4.15 

1.58 

.004884 

Samples  j| 

Alternative 

1 

2 

3 

4 

5 

6 

1 

5.62 

4.94 

7.50 

1.38 

5.94 

2.73 

2 

5.62 

4.94 

7.50 

1.38 

5.94 

2.73 

3 

sssssrsB&ssrsssassa 

5.62 

4.94 

7.50 

1.38 

5.94 

2.73 

Sam; 

Dies 

Alternative 

7 

8 

9 

10 

11 

12 

1 

4.59 

4.77 

3.55 

2.95 

1.57 

2.54 

2 

4.59 

4.77 

3.55 

2.95 

1.57 

2.54 

3 

4.59 

4.77 

3.55 

2.95 

1.57 

2.54 

56 


Upgrade  Combination  5 


code  100 


Alternative 


Samples 

13 

14 

15 

16 

17 

18 

3.10 

2.83 

4.44 

6.19 

6.85 

1.76 

3.10 

2.83 

4.44 

6.19 

6.85 

1.76 

3.10 

2.83 

4.44 

6.19 

6.85 

1.76 

Alternative 


Samples 


19 

20 

21 

22 

23 

24 

2.88 

5.15 

3.68 

4.14 

2.38 

5.49 

2.88 

5.15 

3.68 

4.14 

2.38 

5.49 

2.88 

5.15 

3.68 

4.14 

2.38 

5.49 

Samples 

25 

26 

27 

28 

29 

30 

3.25 

3.08 

5.57 

5.85 

5.24 

4.46 

3.25 

3.08 

5.57 

5.85 

5.24 

4.46 

3.25 

3.08 

5.57 

5.85 

5.24 

4.46 

Alternative 


Tyk  TABLES 


UC5 


Alternative 


Samples 


m 

2 

3 

4 

5 

6 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

Group 

Totals 


UC  5 


Alternative 


UC  5 


Alternative 


B 

8 

9 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

Samples 


10 

11 

12 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

.33 

Group 

Totals 


Samples 

Group 

13 

14 

15 

16 

17 

18 

Totals 

.33 

.33 

.33 

.33 

.33 

.33 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

Samples 

Group 

19 

20 

21 

22 

23 

24 

Totals 

.33 

.33 

.33 

.33 

.33 

.33 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

UC  5 


Alternative 


UC5 

Samples 

Group 

Totals 

Alternative 

25 

26 

27 

28 

29 

30 

1 

.33 

.33 

.33 

.33 

.33 

.33 

2 

2 

.33 

.33 

.33 

.33 

.33 

.33 

2 

3 

.33 

.33 

.33 

.33 

.33 

.33 

2 

Ay,  TABLE 


UC5 

Alternatives 

Group 

1 

2 

3 

Total 

1 

2 

2 

2 

6 

2 

2 

2 

2 

6 

3 

2 

2 

2 

6 

4 

2 

2 

2 

6 

5 

2 

2 

2 

6 

59 


UPGRADE  COMBINATION  6  RAW  DATA 


Upgrade  Combination  6 


code  101 


Grand  Mean 

Standard 

Deviation 

Auto 

Correlation 

4.21 

1.63 

.001857 

4.50 

1.53 

.023120 

4.40 

1.53 

.004219 

Alternative 


Samples 


Samples 

7 

8 

9 

10 

11 

12 

4.59 

4.91 

3.54 

2.94 

1.67 

2.69 

4.87 

5.62 

4.84 

2.40 

1.57 

2.54 

4.86 

6.70 

4.68 

2.74 

1.12 

3.66 

Alternative 


Upgrade  Combination  6 


code  101 


Alternative 


Samples 

13 

14 

15 

16 

17 

18 

3.08 

3.28 

4.44 

6.33 

6.34 

1.76 

3.10 

3.29 

4.55 

6.29 

6.80 

1.64 

2.52 

3.49 

5.02 

5.60 

5.44 

1.96 

Alternative 


Samples 


19 

20 

21 

22 

23 

24 

2.70 

5.55 

2.45 

4.46 

2.36 

5.48 

3.55 

5.15 

3.50 

5.46 

2.38 

5.35 

3.87 

5.52 

3.91 

5.97 

3.12 

5.24 

Samples 

25 

26 

27 

28 

29 

30 

3.24 

2.86 

5.89 

6.73 

5.30 

4.09 

4.31 

3.42 

6.00 

6.31 

6.06 

5.51 

3.30 

3.37 

2.91 

6.73 

5.30 

4.09 

Alternative 


6 


TABLES 


62 


Ay,  TABLE 


UC6 

Alternatives 

Group 

1 

2 

3 

Total 

1 

2 

3 

1 

6 

2 

3 

2 

1 

6 

3 

2 

1 

3 

6 

4 

4 

1 

1 

6 

5 

3 

1 

2 

6 

63 


UPGRADE  COMBINATION  7  RAW  DATA 


Upgrade  Combination  7 


code  110 


Grand  Mean 

Standard 

Deviation 

Auto 

Correlation 

4.59 

1.47 

.001120 

4.76 

1.47 

.004872 

4.68 

1.48 

.003350 

Alternative 


Samples 


Alternative 


Samples 

7 

8 

9 

10 

11 

12 

5.60 

4.32 

3.01 

2.22 

3.10 

3.48 

5.26 

7.16 

4.06 

3.78 

3.03 

4.28 

5.52 

5.41 

3.66 

3.24 

1.59 

4.13 

Upgrade  Combination  7 


code  110 


Alternative 


Samples 


13 

14 

15 

16 

17 

18 

4.27 

4.66 

6.23 

6.18 

2.37 

3.85 

3.67 

3.16 

3.93 

5.83 

6.89 

1.89 

4.59 

3.81 

5.88 

5.24 

7.07 

2.25 

Samples 

19 

20 

21 

22 

23 

24 

5.04 

4.59 

4.66 

2.70 

5.53 

3.49 

4.18 

5.71 

4.71 

4.82 

2.85 

5.87 

5.84 

5.77 

4.35 

5.35 

2.40 

6.36 

Alternative 


Alternative 


Samples 

25 

26 

27 

28 

29 

30 

4.80 

7.16 

6.57 

6.10 

4.64 

3.23 

3.88 

3.78 

3.48 

5.98 

6.48 

4.95 

3.32 

3.08 

6.53 

6.89 

5.63 

4.46 

TABLES 


UC7 

Samples 

Group 

Totals 

Alternative 

B 

2 

3 

4 

5 

6 

1 

1 

0 

B 

1 

0 

0 

3 

2 

0 

0 

0 

0 

0 

0 

0 

3 

0 

1 

0 

0 

1 

B 

3 

UC7 

Samples 

Group 

Totals 

Alternative 

B 

8 

9 

10 

11 

12 

1 

0 

1 

1 

1 

0 

B 

4 

2 

B 

0 

0 

0 

0 

0 

1 

3 

0 

0 

0 

0 

1 

0 

1 

UC7 

Samples 

Group 

Totals 

Alternative 

13 

14 

15 

16 

17 

18 

1 

0 

0 

0 

0 

1 

0 

1 

2 

1 

1 

B 

0 

0 

B 

4 

3 

0 

0 

0 

1 

0 

0 

1 

UC7 

Samples 

Group 

Totals 

Alternative 

19 

20 

21 

22 

23 

24 

1 

0 

1 

0 

1 

0 

B 

3 

2 

1 

0 

0 

0 

0 

0 

1 

3 

0 

0 

1 

0 

1 

0 

2 

UC  7 

Samples 

Group 

Totals 

Alternative 

25 

26 

27 

28 

29 

30 

1 

0 

0 

0 

0 

1 

B 

2 

2 

0 

0 

B 

0 

0 

0 

1 

3 

B 

1 

0 

1 

0 

0 

3 

Ay,  TABLE 


UC  7 

Alternatives 

Group 

1 

2 

3 

Total 

1 

3 

0 

3 

6 

2 

4 

1 

1 

6 

3 

1 

4 

1 

6 

4 

3 

1 

2 

6 

5 

2 

1 

3 

6 

67 


UPGRADE  COMBINATION  8  RAW  DATA 


Upgrade  Combination  8 


code  111 


Grand  Mean 

Standard 

Deviation 

Auto 

Correlation 

4.68 

1.56 

.001330 

5.13 

1.47 

.003980 

4.87 

1.56 

.008338 

Alternative 


Samples 


1 

2 

3 

4 

5 

6 

6.78 

6.05 

7.09 

1.72 

6.58 

3.50 

6.37 

5.12 

7.84 

5.10 

7.27 

3.53 

7.16 

5.75 

7.44 

3.48 

6.22 

4.15 

Alternative 


Samples 

7 

8 

9 

10 

11 

12 

3.30 

5.97 

5.02 

3.45 

2.20 

4.28 

5.50 

6.34 

5.37 

3.96 

2.15 

4.26 

5.80 

6.49 

5.79 

4.58 

3.11 

1.19 

Upgrade  Combination  8 


code  111 


Alternative 


Samples 


13 

14 

15 

16 

17 

18 

2.82 

4.73 

4.65 

6.37 

6.73 

2.23 

3.71 

3.79 

6.27 

6.73 

6.84 

1.89 

4.13 

4.60 

4.00 

6.41 

7.12 

2.24 

Samples 

19 

20 

21 

22 

23 

24 

3.85 

5.41 

4.68 

6.09 

2.70 

5.87 

4.03 

5.71 

4.88 

6.48 

2.85 

5.95 

4.57 

5.74 

5.17 

5.61 

3.13 

6.66 

Alternative 


Samples 

25 

26 

27 

28 

29 

30 

3.05 

3.90 

7.16 

5.56 

4.07 

4.75 

4.34 

4.03 

7.01 

5.94 

5.79 

4.94 

4.20 

3.37 

2.80 

6.75 

4.37 

4.07 

Alternative 


70 


UC  8 

Samples 

Group 

Totals 

Alternative 

25 

206 

27 

28 

29 

30 

1 

B 

0 

0 

1 

1 

0 

3 

2 

0 

0 

0 

0 

0 

0 

0 

3 

0 

1 

1 

0 

0 

B 

3 

Ay,  TABLE 


UC8 

Alternatives 

Group 

1 

2 

3 

Total 

1 

3 

2 

1 

6 

2 

4 

1 

1 

6 

3 

3 

2 

1 

6 

4 

5 

0 

1 

6 

5 

3 

0 

3 

6 

71 


APPENDIX  B;  INTERACTION  PLOTS 


Alot  of  ifftOTActlem  far  yTVf  by 
Parlfthobi Ifty 


0  1 

Piriftrwt>nrty 


74 
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Attn:  Dr.  Brandenstein 

Marine  Corps  Combat  Development  Command 
Quantico,  Virginia  22134-5001 

5.  HQ  TRADOC  2 

Attn:  ATRN-P  CPT  C.A  Chase 

Ft.  Monroe,  Virginia  23651-5000 

6.  Prof  William  G.  Kemple  2 

Code  OR-KE 

Naval  Postgraduate  School 
Monterey,  California  93943-5000 


7.  Prof  Michael  P.  Bailey  1 

Code  OR-BA 

Naval  Postgraduate  School 
Monterey,  California  93943-5000 

8.  Prof  Michael  Sovereign  2 

Code  OR-SM 

Naval  Postgraduate  School 
Monterey,  California  93943-5000 
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